Skip to content

[SPARK-8950][WebUI] Correct the calculation of SchedulerDelay in StagePage #7319

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
carsonwang wants to merge 2 commits into from
Closed

[SPARK-8950][WebUI] Correct the calculation of SchedulerDelay in StagePage #7319

carsonwang wants to merge 2 commits into from

Conversation

@carsonwang
Copy link
Contributor

@carsonwang carsonwang commented Jul 9, 2015

In StagePage, the SchedulerDelay is calculated as totalExecutionTime - executorRunTime - executorOverhead - gettingResultTime.
But the totalExecutionTime is calculated in the way that doesn't include the gettingResultTime.

@SparkQA
Copy link

SparkQA commented Jul 9, 2015

Test build #36913 has finished for PR 7319 at commit 43021f6.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

srowen
srowen reviewed Jul 9, 2015
View reviewed changes
core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala Outdated
Copy link
Member

@srowen srowen Jul 9, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, is it right to make this 0 before the job is finished? that wasn't the current behavior. What are you trying to make this consistent with?

Copy link
Contributor

@kayousterhout kayousterhout Jul 10, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is the current behavior? If the job isn't finished, totalExecutionTime will be 0, and max(0, 0 - some stuff) = 0?

Copy link
Contributor Author

@carsonwang carsonwang Jul 10, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the current result of SchedulerDealy is also 0 for runing tasks. The metrics.executorRunTime is 0 when the task is still running.

@andrewor14
Copy link
Contributor

andrewor14 commented Jul 9, 2015

@kayousterhout

kayousterhout
kayousterhout reviewed Jul 10, 2015
View reviewed changes
core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala Outdated
Copy link
Contributor

@kayousterhout kayousterhout Jul 10, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you use the old line break here? This isn't consistent with the style guide (and easier to read when neither term has a line break in the middle)

Copy link
Contributor Author

@carsonwang carsonwang Jul 10, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. I'll add the line breaks.

@SparkQA
Copy link

SparkQA commented Jul 10, 2015

Test build #36988 has finished for PR 7319 at commit 8cb6ece.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@carsonwang
Copy link
Contributor Author

carsonwang commented Jul 13, 2015

Jenkins, retest this please.

@SparkQA
Copy link

SparkQA commented Jul 13, 2015

Test build #37108 has finished for PR 7319 at commit 8cb6ece.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@carsonwang carsonwang force-pushed the SchedulerDelayTime branch from 8cb6ece to f66fb6e Compare July 13, 2015 05:20
@SparkQA
Copy link

SparkQA commented Jul 13, 2015

Test build #37125 has finished for PR 7319 at commit f66fb6e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@kayousterhout
Copy link
Contributor

kayousterhout commented Jul 13, 2015

LGTM

carsonwang added a commit that referenced this pull request Jul 13, 2015
@carsonwang @kayousterhout
[SPARK-8950] [WEBUI] Correct the calculation of SchedulerDelay in Sta…
5ca26fb 
…gePage

In StagePage, the SchedulerDelay is calculated as totalExecutionTime - executorRunTime - executorOverhead - gettingResultTime.
But the totalExecutionTime is calculated in the way that doesn't include the gettingResultTime.

Author: Carson Wang <[email protected]>

Closes #7319 from carsonwang/SchedulerDelayTime and squashes the following commits:

f66fb6e [Carson Wang] Update the code style
7d971ae [Carson Wang] Correct the calculation of SchedulerDelay
@kayousterhout
Copy link
Contributor

kayousterhout commented Jul 15, 2015

@carsonwang would you mind closing this? For some reason, it wasn't closed automatically when I merged it into master.

@carsonwang
Copy link
Contributor Author

carsonwang commented Jul 16, 2015

Sure. Thanks @kayousterhout

@carsonwang carsonwang closed this Jul 16, 2015
@carsonwang carsonwang deleted the SchedulerDelayTime branch August 17, 2015 01:23
mingyukim pushed a commit to palantir/spark that referenced this pull request Sep 22, 2015
@carsonwang @mingyukim
[SPARK-8950] [WEBUI] Correct the calculation of SchedulerDelay in Sta…
ae800b7 
…gePage

In StagePage, the SchedulerDelay is calculated as totalExecutionTime - executorRunTime - executorOverhead - gettingResultTime.
But the totalExecutionTime is calculated in the way that doesn't include the gettingResultTime.

Author: Carson Wang <[email protected]>

Closes apache#7319 from carsonwang/SchedulerDelayTime and squashes the following commits:

f66fb6e [Carson Wang] Update the code style
7d971ae [Carson Wang] Correct the calculation of SchedulerDelay
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@carsonwang @SparkQA @andrewor14 @kayousterhout @srowen